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EXTENDING  THE  TRAINING  EFFICIENCY  AND  EFFECTIVENESS 
METHODOLOGY  (TEEM)  WITH  TRAINING  TRANSFER  DATA 

SUMMARY 

The  Training  Efficiency  and  Effectiveness  Methodolgy  (TEEM)  was  extended  using 
training  transfer  data.  Coxirse  efficiency,  field  effectiveness  and  transfer  data  were  integrated  in  an 
application  to  an  Aerospace  Ground  Equipment  technical  training  course  to  demonstrate  how 
these  data  could  be  utilized  for  training  system  re-design. 

L  INTRODUCTION 

A  significant  amount  of  resources  are  devoted  to  training  in  the  private  sector 
(Tannenbaum  &  Yukl,  1992)  as  well  as  in  the  military  (Ruck,  Thompson,  &  Stacy,  1987). 
Therefore,  training  that  is  irrelevant,  inefficient,  or  ineffective  can  be  costly.  Due  to 
organizational  downsizing,  the  associated  reduction  of  resources  devoted  to  training,  and  the 
increased  complexity  of  technology,  equipment  and  jobs  (Howell  &  Cooke,  1988),  improving  the 
quality  of  training  has  never  been  more  critical.  The  purpose  of  this  paper  is  to  describe  an  effort 
directed  at  assessing  and  utilizing  training  efficiency,  trmning  effectiveness  and  training  transfer 
information  for  training  system  improvement. 

Background 

The  most  popular  framework  for  evaluating  trmning  programs  was  developed  by 
Kirkpatrick  (1959;  1960).  This  four-level  framework  includes:  student  reactions  to  training; 
learning  of  facts,  principles  or  techniques;  changes  in  on-the-job  behavior;  and  tangible  business 
results.  Although  this  framework  has  advanced  the  practice  of  training  evaluation  over  the  years, 
it  does  not  offer  a  systematic  approach  for  the  use  of  evaluation  information  to  improve  training. 

The  systems  approach  to  trmning  specifies  the  use  of  evaluation  information  for  training 
system  improvement.  In  fact,  evaluation  is  depicted  as  a  key  component  that  provides  feedback 
information  to  the  needs  assessment,  design,  development  and  deliveiy  stages  of  the  process  to 
continuously  update  and  improve  training  quality  (Goldstein,  1993).  Thus,  evaluation  information 
should  be  instrumental  in  helping  human  resource  managers  and  training  personnel  make 
informed,  accurate  and  meaningful  changes  to  training.  Unfortunately,  training  evaluation 
information  is  typically  collected  to  determine  trainee  reactions  to  the  course,  \vith  less  attention 
pmd  to  whether  learning  objectives  were  met,  and  almost  no  attention  given  to  subsequent 
transfer  of  training  and  performance  on  the  job  (Alliger  &  Janak,  1989;  Baldwin  &  Ford,  1988). 
As  a  result,  evaluation  information  is  rarely  available  or  utilized  to  improve  training,  and 
subsequently,  little  research  has  been  conducted  in  this  area. 

One  exception  is  Ford  and  Wroten  (1984),  who  developed  a  Matching  Technique  that 
compared  training  needs  with  current  training  emphasis  in  order  to  link  training  evaluation  to 
trmning  needs  assessment  and  facilitate  course  re-design.  Ford  and  Sego  (1990)  recommended 
that  this  technique  be  used  to  determine  training  efficiency,  that  is,  the  extent  to  which  tasks  may 
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be  over-  or  under-trained.  They  extended  the  Matching  Technique  by  developing  a  conceptual 
framework  to  integrate  efficiency  and  effectiveness  information  for  course  re-design.  Since  then, 
efforts  have  demonstrated  the  use  of  the  matching  technique  to  determine  training  efficiency  and 
make  course  rewsions  (Teachout,  Olea,  Phalen,  &  Barham,  1993)  as  well  as  integrating  efficiency 
and  course  effectiveness  information  for  course  re-design  (Teachout,  Sego,  &  Ford,  1995). 

While  these  efforts  have  advanced  the  use  of  evaluation  data  for  course  revision,  they  did 
not  incorporate  information  from  the  job  context.  The  importance  of  training  transfer  to  the  job 
has  been  well  documented  (e.g.,  Baldwin  &  Ford,  1988).  An  important  factor  in  facilitating  the 
transfer  of  skills  is  the  trainee's  opportunity  to  perform  the  trained  tasks  on-the-job.  Ford, 
Quinones,  Sego  and  Sorra  (1992)  demonstrated  that  individuals  obtained  relatively  different 
opportunities  to  perform  (OTP)  trained  tasks.  Since  the  lack  of  opportunity  to  perform  tasks  is 
related  to  performance  decrements  (Fendrich,  Healy,  Meiskey,  Crutcher,  Little,  &  Borne,  1988), 
on-the-job  task  performance  of  trainees  should  be  higher  for  tasks  that  receive  relatively  more 
OTP.  Ford  et  al.  (1992)  suggest  the  need  for  integrating  training  program  content  with  on-the- 
job  opportunities  to  increase  the  understanding  of  training  transfer  and  training  effectiveness. 

The  present  paper  extends  previous  work  by  Teachout,  Sego  and  Ford  (1995)  by  incorporating 
training  transfer  and  job  effectiveness  data  into  Ford  and  Wroten's  (1984)  matching  technique, 
and  demonstrating  how  the  combination  of  efficiency,  effectiveness  and  transfer  data  can  be  used 
for  training  system  re-design. 


n.  METHOD  AND  PROCEDURE 
Training  Course  and  Job  Context 

This  study  integrated  training  efficiency,  trmning  transfer  and  training  effectiveness 
information  from  a  United  States  Air  Force  Aerospace  Ground  Equipment  (AGE)  Airman  Basic- 
in-Residence  (ABR)  technical  training  course  and  subsequent  on-the-job  information.  The  AGE 
ABR  course  consists  of  18  weeks  of  instruction  regarding  powered  and  non-powered  equipment 
used  to  support  aircraft  in  the  Air  Force. 

Participants 

Participants  were  182  Aerospace  Ground  Equipment  (AGE)  trainees  in  the  Airman  Basic- 
in-Residence  (ABR)  course  at  Chanute  AFB,  IL.  In  addition,  the  training  manager,  course 
developers  and  instructors  participated  in  specifying  course  content.  Finally,  182  supervisors  of 
the  training  graduates  provided  on-the-job  performance  ratings  for  the  trained  tasks. 

Training  Efficiency 

Ford  and  Wroten's  (1984)  Matching  Technique  was  used  to  determine  training  efficiency. 
Training  efficiency  examines  whether  the  appropriate  amount  of  time  is  devoted  to  each  task 
taught  in  training,  or  whether  tasks  are  potentially  over-  or  under-trained.  This  requires  trdning 
needs  information  that  indicates  the  relative  time  required  to  learn  a  task  and  the  actual  time 
devoted  to  training  each  task.  Occupational  Survey  Reports  (OSRs)  produced  by  the  Air  Force's 
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occupational  measurement  process  contdn  a  rich  source  of  information  that  is  used  to  identify  and 
prioritize  tasks  to  be  included  in  initial  technical  training  courses.  From  the  OSR,  the  task 
learning  difficulty  rating  was  used  as  the  measure  of  the  time  to  learn  a  task.  Since  this  index  is 
based  on  the  notion  of  task  learning  time,  it  provides  a  reasonable  estimate  of  the  relative  time 
that  should  b_e  devoted  to  training  each  task.  The  construct  validity  of  the  task  learning  difficulty 
ratings  have  been  well  established  against  entry-level  training  variables  (Mumford,  Weeks, 
Harding,  &  Fleishman,  1987)  and  job  performance  measures  (Dickinson  &  Teachout,  1993). 

A  four-step  procedure  was  developed  to  collect  information  on  the  actual  time  (in  hours) 
devoted  to  each  task  taught  in  training.  First,  course  personnel  determined  that  99  OSR  tasks 
were  taught  in  the  course.  Second,  course  instructors  linked  these  99  tasks  to  specifc  training 
objectives  in  the  course  Plan  of  Instruction  (POI).  In  several  cases,  a  single  task  was  covered  in 
more  than  one  training  objective.  Third,  instructors  divided  the  time  allocated  for  each  training 
objective  among  all  of  the  tasks  covered  in  the  objective.  For  example,  if  an  objective  contains  6 
hours  of  instruction  and  3  tasks  are  covered  in  that  objective,  the  instructors  estimated  how  much 
of  the  6  hours  was  spent  on  instruction  for  each  task.  Fourth,  the  total  amount  of  training  time 
devoted  to  each  task  was  summed.  Since  tasks  could  be  covered  by  more  than  one  training 
objective,  the  training  time  for  each  task  was  summed  across  the  appropriate  training  objectives  to 
specifiy  the  amount  of  course  time  spent  training  that  task. 

Before  the  task  learning  difficulty  rating  and  the  hours  spent  in  training  were  matched  for 
each  task,  both  variables  were  transformed  into  standard  scores  with  a  mean  of  “0”  and  a  standard 
deviation  of  “1”.  Ford  and  Wroten's  (1984)  Matching  Technique  was  then  used  to  match  these 
two  data  elements. 

Training  Effectiveness 

Training  effectiveness  addresses  whether  trainees  learned  or  can  perform  the  tasks  taught 
in  training.  This  requires  information  about  the  knowledge  or  performance  levels  of  trainees  at 
the  end  of  the  training  program  and  later  on  the  job.  The  greater  the  learning  and  performance, 
compared  to  a  specified  standard,  the  greater  the  effectiveness  of  the  training  program.  In  this 
study,  supervisor  ratings  of  task  performance  were  provided  after  trainees  had  worked  for  8 
months  on-the-job.  Thirty-three  of  the  99  tasks  were  sampled  to  represent  different  content  areas 
(i.e.,  AGE  equipment)  across  different  levels  of  task  learning  difficulty  (Ford  et.  al.,  1992).  A  7- 
point  Likert-type  scale  ranging  from  1 -poor,  or  well  below  acceptable  level  of  performance,  to  7- 
excellent,  or  well  above  acceptable  level  of  performance,  was  used.  The  mid-point  of  the  scale,  4- 
average,  or  meets  acceptable  level  of  performance,  was  considered  the  standard  to  illustrate 
whether  performance  for  a  given  task  fell  above  or  below  that  standard.  The  mean  performance 
rating  across  all  incumbents  (i.e.,  training  graduates)  was  calculated  for  each  of  the  33  tasks  to 
represent  training  effectiveness.  If  the  mean  rating  was  4.00  or  greater,  training  was  considered 
effective  since  graduates  were  performing  above  standard  .  If  the  mean  rating  was  less  than  4.00 
training  was  considered  ineffective  since  graduates  were  performing  below  standard. 
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Training  Transfer 

The  OTP  trained  tasks  measure  (Ford  et  al.,  1992)  was  used  as  the  measure  of  training 
transfer.  Each  trainee  recorded  the  number  of  times  they  had  performed  each  of  the  33  tasks  in 
the  8  month  j)eriod  since  they  had  arrived  on-the-job.  The  mean  number  of  times  performed 
(NTP)  across  all  incumbents  was  calculated  for  each  of  the  33  tasks  to  represent  training  transfer. 

m.  RESULTS 


Training  Efficiency 

Figure  1  summarizes  the  results  of  the  matching  technique  for  the  33  tasks.  The  task 
learning  difficulty  rating  (i.e.,  relative  time  to  learn)  is  represented  on  the  x-axis  and  the  actual 
ourse  time  spent  trmning  each  task  is  represented  on  the  y-axis.  Three  potential  outcomes  are 
depicted.  Training  matches,  depicted  as  the  area  between  the  diagonal  lines,  result  when  the 
difference  between  the  two  variables  is  small  (i.e.,  the  standard  score  difference  is  1  standard 
deviation  or  less).  Fifteen  of  the  33  tasks  (45%)  were  considered  training  matches,  indicating 
that  the  relative  amount  of  time  spent  in  training  seems  appropriate.  Potentially  over-trmned 
tasks,  depicted  in  the  upper  left  of  the  graph,  result  when  relatively  more  time  is  spent  training  a 
task,  compared  to  the  corresponding  task  learning  difficulty  variable  (i.e.,  the  actual  training  time 
standard  score  exceeds  the  task  learning  difficulty  standard  score  by  more  than  1  standard 
deviation).  Twelve  of  the  33  tasks  (36%)  were  considered  potentially  over-trained  tasks, 
indicating  that  training  time  might  be  reduced  for  these  tasks.  Potentially  under-trained  tasks, 
depicted  in  the  bottom  right  of  the  graph,  result  when  the  task  learning  difficulty  variable  is 
relatively  greater  than  the  corresponding  actual  training  time  variable  (i.e.,  the  task  learning 
difficulty  standard  score  exceeds  the  actual  task  time  standard  score  by  more  than  1  standard 
deviation).  Six  of  the  33  tasks  (18%)  were  considered  potentially  under-trained  tasks  indicating, 
that  more  training  time  might  be  allocated  to  these  tasks. 


ACTUAL  COURSE 
EMPHASIS 

(TIME  IN  HOURS) 


OSR  TASK  LEARNING  DIFFICULTY 
(TIME  TO  LEARTO 

Figure  1.  Matching  Technique  to  Assess  Training  Efficiency. 
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Training  Efficiency,  Effectiveness  and  Transfer  Integration 

Figure  2  summarizes  the  training  efficiency,  effectivenss  and  transfer  integration  in  a  3  X 
2  matrix.  The  three  rows  represent  training  efficiency  in  the  form  of  the  three  potential  outcomes 
of  the  Matching  Technique.  The  two  columns  represent  training  effectiveness  by  depicting  task 
performance  as  above  or  below  standard.  Training  transfer  is  repesented  by  the  mean  NTP 
associated  with  the  tasks  in  each  box  of  the  matrix. 

Twenty-nine  of  the  33  tasks  were  performed  above  standard.  The  average  performance 
level  is  highest  for  the  over-trained  tasks  and  lowest  for  the  undertrained  tasks.  This  is  consistent 
with  expectations,  since  more  training  time  relative  to  the  recommended  time  (i.e.,  over-trained 
tasks)  results  in  higher  performance  levels,  while  less  training  time  relative  to  the  recommended 
time  results  in  lower  performance  levels. 

The  average  NTP  is  highest  for  over-trained  tasks  and  lowest  for  under-trained  tasks. 

This  is  also  consistent  with  expectations.  Tasks  that  receive  more  time  in  training  are  typically 
performed  more  fi-equently  on-the-job,  since  the  percentage  of  incumbents  performing  a  task  on- 
the-job  is  a  factor  used  in  prioritizing  tasks  for  inclusion  in  Air  Force  technical  training 
programs.  In  this  study,  tasks  that  receive  more  training  time  relative  to  the  recommended  time 
from  the  Matching  Technique  are  also  performed  more  frequently  for  this  sample  as  indicated  by 
the  NTP  results. 

The  mean  NTPs  are  also  much  lower  for  the  tasks  performed  below  standard  compared 
to  the  tasks  performed  above  standard.  Regardless  of  the  efficiency  outcome,  this  indicates  that 
NTP  may  be  a  significant  factor  in  improving  on-the-job  task  performance. 

Figure  2.  Training  Efficiency,  Effectiveness  and  Transfer  Integration 
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IV.  DISCUSSION  AND  SUMMARY 


This  paper  extended  previous  work  by  demonstrating  the  potential  use  of  training 
efficiency,  training  effectiveness  and  training  transfer  information  for  trmning  system  re-design. 
Ford  &  Wroten  (1984)  developed  and  demonstrated  the  use  of  the  Matching  Technique  for 
training  program  re-design.  Teachout,  Olea,  Barham,  &  Phalen  (1993)  applied  this  technique  to 
determine  training  efficiency  and  facilitate  course  revisions,  while  Teachout,  Sego  &  Ford 
(1995)  integrated  training  efficiency  and  course  effectiveness  data  for  course  re-design.  The 
present  study  includes  the  use  of  job  effectiveness  and  trmning  transfer  data  that  permits  a 
broader  application  beyond  the  scope  of  the  course  to  trmning  system  re-design. 

Examined  separately,  the  results  of  training  efficiency,  training  effectiveness  and  training 
transfer  data  provide  useful  information  for  understanding  and  changing  different  aspects  of  a 
training  system.  Results  of  the  Matching  Technique  suggest  that  the  course  efficiency  can  be 
improved  by  reducing  training  time  for  over-trained  tasks  and  increasing  trmning  time  for  under¬ 
trained  tasks.  Training  effectiveness  data  suggest  that  for  tasks  performed  below  the  performance 
standard,  course  revisions  may  be  necessary,  or  perhaps  required,  while  tasks  performed  above 
standard  are  less  likely  to  require  course  changes.  Training  transfer  data  indicates  that  more  task 
experience  might  be  required  early  on-the  job  to  facilitate  job  performance  where  the  opportunity 
to  perform  the  trdned  tasks  is  low.  However,  these  data  are  more  useful  when  considered  in 
combination.  For  example,  the  results  illustrate  how  a  training  program  can  be  re-designed  to  be 
more  efficient  by  reducing  training  time  for  over-trained  tasks  that  were  performed  above 
standard,  while  increasing  training  time  for  under-trained  tasks  that  were  performed  below 
standard.  Furthermore,  the  training  transfer  information  is  useful  in  determining  whether  the 
training  program  itself  should  be  modified,  or  the  transfer  environment  changed.  For  example,  if  a 
task  is  under-trained,  the  amount  of  practice  on  the  job  is  sufficient,  but  it  is  performed  below 
standard,  the  traning  program  might  be  modified  by  increasing  the  training  time  for  that  task. 
Alternatively,  if  an  over-trained  task  is  performed  below  standard  on-the-job,  yet  the  amount  of 
practice  for  that  task  is  low,  modifying  the  training  program  is  probably  not  a  reasonable  action. 

A  better  solution  might  be  to  ensure  training  graduates  receive  more  practice  for  that  task  on  the 
job,  or  consider  eliminating  it  from  training,  and  allocate  that  time  to  other  tasks  in  the  program. 
Of  course,  other  options  should  always  be  considered,  such  as  developing  improved  instructional 
approaches. 

In  the  Air  Force,  training  personnel  frequently  make  training  course  rewsions  due  to 
reductions  in  course  length,  additional  requirements  dictated  by  policy  changes,  or  customer 
feedback  that  suggests  that  an  area  of  trmning  is  deficient.  This  methodology  utilizes  relevant 
information,  integrates  that  information  in  a  way  that  is  easy  to  use  and  understand,  and  puts  it  in 
a  format  that  can  facilitate  decision-making  by  training  course  personnel. 


6 


In  summary,  this  paper  illustrates  an  extension  of  TEEM  by  integrating  training  efBciency, 
effectiveness  and  transfer  iiiformation  and  displajing  the  information  in  a  format  that  facilitates 
training  course  revisions.  Conceptual  models  of  trmning  (e.g.,  Goldstein,  1993)  suggest  that 
evaluation  plays  a  vital  role  in  helping  training  personnel  re-assess  training  needs  and  re-allocate 
scarce  training  resources.  This  methodology  helps  us  take  a  step  in  that  direction.  While  each 
piece  of  information  is  useful  in  evaluating  different  aspects  of  a  training  system,  considered  in 
combination,  training  efficiency,  effectiveness  and  transfer  provide  powerful  diagnostic 
information  for  re-designing  a  training  system. 
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